LIMITS OF BC TYPE ORTHOGONAL POLYNOMIALS AS 
THE NUMBER OF VARIABLES GOES TO INFINITY 



Andrei Okounkov and Grigori Olshanski 



Abstract. We describe the asymptotic behavior of the multivariate BC— type Ja- 
cobi polynomials as the number of variables and the Young diagram indexing the 
polynomial go to infinity. In particular, our results describe the approximation of 
the spherical functions of the infinite-dimensional symmetric spaces of type B, C, D 
or BC by the spherical functions of the corresponding finite-dimensional symmetric 
spaces. Similar results for the Jack polynomials were establishe d in our earlier paper 
(Intern. Math. Res. Notices 1998, no. 13, 641-682; arXiv: j-alg/9709011 ) . The 
main results of the present paper were obtained in 1997. 



Contents 

1. Introduction 

IA. BC n orthogonal polynomials 

IB. Statement of the main result 

IC. Other results 

2. Interpolation BC n polynomials and binomial formula 
2A. Interpolation BC n polynomials 

2B. Binomial formula 

2C. Asymptotics of denominators in binomial formula 

3. Sufficient conditions of regularity 

4. Necessary conditions of regularity 

5. The convex set T e 

6. Spherical functions on infinite-dimensional symmetric spaces 

7. The BC n polynomials with 6=1 
References 



1. Introduction 

In this paper we evaluate the limit behavior of the (suitably normalized) multi- 
variate orthogonal polynomials associated to the root system BC n as n — > oo. For 
given n = 1, 2, ... , these polynomials are viewed as functions on the n-dimensional 
torus T" . They are indexed by an arbitrary partition A of length at most n, and also 
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depend on 3 parameters 9 > 0, a > — 1, b > — 1. The normalization is determined 
by the condition that the polynomials take value 1 at the point (1, . . . , 1). 

We let n go to infinity and assume that the partition A = A(n) varies together 
with n. We obtain necessary and sufficient conditions on the sequence {A(n)} under 
which the corresponding polynomials uniformly converge on any fixed subtorus T fc , 
k = 1, 2, We also describe all possible limit functions, which live on an infinite- 
dimensional torus; it turns out that they depend on countably many continuous 
parameters. 

The motivation for studying this asymptotic problem comes from representa- 
tion theory of infinite-dimensional classical groups. For certain special values of 
the parameters 9, a, b (in particular, has to be one of the numbers 5, 1, 2) the 
normalized orthogonal polynomials of type BC n can be interpreted as the inde- 
composable spherical functions on rank n symmetric spaces of compact type, with 
restricted root system B n , C n , D n or BC n . Likewise, the limit functions can be 
interpreted as indecomposable spherical functions on certain infinite-dimensional 
analogs of these symmetric spaces. 

Our results not only provide a complete classification of the spherical functions on 
infinite-dimensional symmetric spaces but also explain how these "infinite-variate" 
spherical functions are approximated by the conventional ( "finite-variate" ) spheri- 
cal functions. 

The present paper can be viewed as a continuation of our paper [004] where we 
studied a similar asymptotic problem for the Jack polynomials. The Jack polyno- 
mials depend on a single continuous parameter > 0. For 3 special values |, 1, 2 
of this parameter, the (suitably normalized) Jack polynomials with n variables can 
be interpreted as indecomposable spherical functions on rank n compact symmetric 
spaces with restricted root system A n : 

U(n)/0(n), (U(n) x U(n))/U(n), U(2n)/Sp(n). 

In the case 9 = 1 the Jack polynomials become the Schur polynomials. The 
normalized Schur polynomials can also be viewed as the normalized irreducible 
characters of the groups U(n). The large n asymptotics of these characters was 
found, for the first time, in the pioneer work of A. M. Vershik and S. V. Kerov, 
[VK]. 

The results of our paper [004] and of the present paper together provide a far 
generalization of [VK] , involving all families of multivariate orthogonal polynomials 
connected with classical root systems. A natural field of application for our results is 
infinite-dimensional noncommutative harmonic analysis in the spirit of the papers 
Olshanski [04] and Borodin-Olshanski [BorO]. Those papers present a detailed 
study of harmonic analysis on the infinite-dimensional unitary group, and they 
substantially use the large n asymptotics of Schur polynomials. One can expect 
that a similar theory can be built for other infinite-dimensional classical groups 
or symmetric spaces, which will necessarily imply a similar use of more general 
orthogonal polynomials. 

We proceed to a more detailed description of the results obtained in the present 
paper. 

1A. BC n orthogonal polynomials. 

Throughout the paper n = 1,2, .. . denotes a natural number. Let W denote 
the BC n Weyl group 

W = S{n) k Z™ . 
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We shall need two copies W, and W+ of this group acting on functions in n variables. 
The S(n) part in both cases permutes the variables z\,. . . ,z n , and the Z£ part acts 

by 

f(z) i * f(zt\ zt 1 ) , f(z) i * /(±zi, . . . , ±z n ) , 

in M^, and W+, respectively. 

Given a partition A, we denote by £(X) the number of nonzero parts of A. The 
BC n orthogonal polynomials (also called the BC n Jacobi polynomials) are certain 
VK* -invariant Laurent polynomials in n variables, labelled by partitions A with 
£(X) < n and depending on 3 parameters 

0>O, a,b>-l. 

The polynomials are defined as follows. The parameters 6, a, b specify an inner 
product, 

(f,g) = / f(z)~gjzjm(z) • Haar(dz) , 

of functions on the n-dimensional torus 

T n = {(z u ...,z n )cC n }, \zi\ = l, 

where "Haar" is the Haar measure on T™ and tt>(z) is the following M4-invariant 
weight function 

»(*)= n izi-zjni-z^i 29 n ii-z^ii+z^. (i.i) 

\<i<j<n l<i<n 

The polynomials in question, denoted as 3x {z; 0, a, b), are W4-invariant, orthogo- 
nal with respect to the above inner product, and satisfy the triangularity condition 

3 A (z;0,a,6) = z A + ... , (1.2) 

where z x = z\ 1 • • • z^ n and dots stand for lower monomials in the lexicographic 
order. These properties characterize the polynomials 3\(z; 9, a, b) uniquely. 

Actually, the polynomials 3\(z; 6, a, b) possess a stronger triangularity property. 
To state it we need some notation. Given a partition /j, with £{fi) < n, set 

(summed over weights v in the W+ -orbit of fi). This is an analog of the monomial 
symmetric function for the root system BC n . Next, let £\, . . . , e n be the canonical 
basis in Z™. Write /j, <C A if the vector (Ai — /ii,...,A n — [i n ) G Z" can be 
written as a linear combination of the vectors £j — ej (1 < i < j < n) and £j 
(1 < i < n) with nonnegative integral coefficients. In this notation, the refinement 
of the triangularity condition has the form: 

d\{zi, ...,z n ;6,a,b)= ^ ua^(6>, a, b) m^zx, . . .,z n ), (1.4) 



4 



ANDREI OKOUNKOV AND GRIGORI OLSHANSKI 



where u\ t _ l (9, a, b) are certain coefficients such that u\\(9 7 a,b) = 1. 

This is a specialization of the general definition of the multivariate Jacobi poly- 
nomials corresponding to an arbitrary root system R. See, for example, Heckman's 
lectures published in [HS], Macdonald [Ma4], and Koornwinder's expository paper 
[K2]. Those general polynomials depend on a Weyl group invariant function 

a i— » k a , a G R, 

on the root system R. In our case 

R = {±£i ± Sj , ±£ 4 , ±2£j , 

where {si}, as above, is the standard basis of K", and 

k±ei±ej=26, k ±Ei =a-b, k ±26z = b + \ . (1.5) 

The numbers A: Q are viewed as formal root multiplicities. Thus, the "half -sum of 
the positive roots" is defined as 

p := i k » a = ( d ( n - 1) + f, 0(n - 2) + <r, . . . , + ct, ct) , 

where 

a + fe+1 

We shall need the following known fact: 

Proposition 1.1. Assume a > b > — i so i/iai i/ie formal multiplicities defined in 
(1.5) are nonnegative. Then in the expansion (1.4), i/ie coefficients u\ fl (9,a 1 b) are 
all nonnegative. 

Proof. See Macdonald's paper [Ma4], formula (11.15) and the argument following 
it. □ 

By an appropriate specialization of the parameters a, 6 one can obtain the or- 
thogonal polynomials associated to the root systems B ni C n or D n . Namely, we 
have toset6=— i,a = 6^— iora = 6= — |, respectively. It should be noted, 
however, that in the D n case, the polynomial 3\{z', 9, a, b) — 3\{z', 9, — |, — |) with 
A„ > is the sum of certain two "twin" D„-polynomials. 

Note that the orthogonal polynomials for the A n root system are very closely 
related to Jack polynomials (see Beerends and Opdam [BeO]); the asymptotics of 
Jack polynomials asn^oo was studied in our paper [004]. As for the exceptional 
root systems, there are, obviously, nomoo asymptotic problems. 

Similarly to the Jack polynomials, the polynomials d\(z;9,a, b) are eigenfunc- 
tions of n commuting differential operators and describe excitations in certain com- 
pletely integrable quantum many body systems which were introduced by Olshanet- 
sky and Perelomov [OP]. 

For certain special values of the parameters 9, a, b, the root data (R, {k a }) given 
by (1.5) correspond to the restricted root system of a rank n compact symmetric 
space G(n)/K(n) of classical type (for more detail, see §6). Then the commuting 
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differential operators mentioned above become the radial parts of invariant differ- 
ential operators on G(n)/K(n). 

Next, we introduce the normalized Jacobi polynomials 

$ A (z;6»,a,6) := — — — — . 1.6 

d\{l,...,l;0,a, b) 



For special values of 9, a, b these are the indecomposable spherical functions on the 
corresponding symmetric space G{n)/K{n). 

The Jacobi polynomials S\(z;9,a,b) are degenerations as q — > 1 of the 6-pa- 
rametric Koornwinder polynomials which are eigenfunctions of certain commuting 
q-difference operators (see Koornwinder [Kl], van Diejcn [Di]). 

Finally, note that the Jacobi polynomials 3x{z',9,a,b) can be transferred from 
the torus T" to the cube [—1,1]™ via the map 

_ zj + z^_ 

which takes Wi-invariant polynomials in z to symmetric polynomials in x. The 
weight function (1.1) is then replaced by the weight function 

n i^-^-r n (i-^r(i+^) h 

l<i<j<n l<i<n 

with respect to Lebesgue measure dx on the cube. When n = 1, this is the familiar 
weight function for the classical Jacobi polynomials. 

IB. Statement of the main result. 

Fix some 9 > 0. We could have also fixed some a, b > —1; however, at no extra 
cost, we can consider the following more general situation. Namely, we fix two 
sequences 

{a„},{6„}, 
such that a n , b n > — 1 and the limits 

a = lim — , b = lim — (1-7) 

exist. In particular, if a n and b n do not depend on n then 

a = b = 0. 

We consider 6*, {a„}, and {&„} as fixed parameters of our problem and study the 
limit behavior as n — > oo of the functions 

$ X (n)(zi, ■ ■ ■ ,z k , l^^;9,a n ,b n ) , (1.8) 

n — k times 

where k — 1, 2, 3, . . . is fixed and {A(n)} is a sequence of partitions with £(X(n)) < n, 
A(n) - (A(n)i > A(n) 2 > ■ ■ ■ > A(n) n > 0). (1.9) 
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Definition 1.2. Let {X(n)} be a sequence of partitions as in (1.9). 

(i) {A(n)} is said to be regular if for every fixed k, the functions (1.8) uniformly 
converge on the torus T fe , as n — > oo. 

(ii) {A(n)} is said to be infinitesimally regular if for every fixed k, the Tay- 
lor expansions of (1.8) in a local system system of coordinates about the point 
(1, . . . , 1) G T fe have a coefficient -wise limit; in this case we shall say that the func- 
tions (1.8) converge infinitesimally. In other words, this type of convergence means 
the convergence of jets at the point (1, . . . , 1) G T fe . 

(iii) {A(n)} is said to be minimally regular if the functions 

$A(n)(z, h_^A-,0,a n ,b n ) , \z\ = l, 

n—l times 

converge pointwise to a continuous function on T 1 . 

Given a partition A, let A' denote the conjugate partition. That is, A^ is the 
length of the ith column in the diagram of A. Let |A| denote the sum of the parts 
of A (equivalently, the number of the boxes in the corresponding diagram) . 

Definition 1.3. A sequence (1.9) is said to be a Vershik-Kerov sequence (VK 
sequence, for short) if the following limits exist: 



1,2,... , 

= 1,2,..., (1-10) 



on '■= hm 


X(n)i 


< oo , 




n 




/3i := lim 


(A(n)X 


< oo 




n 




5 := lim 


\X(n)\ 


< oo . 



n 

It is readily checked (see below) that the number 

is nonnegative. The numbers Qj, /3,, 7 (or a*, S) are called the VK parameters 
of the sequence {A(n)}. 

Let us give a slightly different (but equivalent) definition of the VK parameters, 
which makes evident the inequality 7 > 0. Let d(n) denote the number of diagonal 
boxes in the Young diagram corresponding to A(n). For i = 1, . . . , d(n), we replace 
in (1.10) the row lengths A(n), and the column lengths (A(n))- by the respective 
modified Frobenius coordinates 

\{n)i-i + \, {\{n))'i-i+\, 

and for i > d(n), we replace the row and column lengths by zeros, which does not 
affect the definition of ai and Pi. On the other hand, the sum of the modified 
Frobenius coordinates equals A(n)|. After the limit transition, this turns into the 
inequality J2i a i + Pi) ^ <^> so tna t 7 > 0. 

The following is the main result of the present paper. 
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Theorem 1.4. Let 9 > and a n > b n > — | be parameters satisfying (1.7). Let 
{A(n)}„=i j 2 1 ... be a sequence of partitions with £(X(n)) < n. 

(i) All 3 regularity properties of Definition 1.2 are equivalent to each other and 
are also equivalent to the VK conditions (1.10) of Definition 1.3. 

(ii) If {A(n)} is a VK sequence with parameters on, 7 then for any fixed 
fc = l,2,... 

JL (z j + zj 1 \ 
Jim o $ A( „ ) (z 1 ,...,z fe , l^^l; 0,a n ,b n ) = [ [ ^ a , 0lJ ,a,b I I . 

n — k times ^~ ^ 

where a,b are defined in (1.7), and <f> a p~ai * s following function of a single 
variable x G [—1,1] 

1_ 2tf ( J 

In particular, if a = b = f w/wc/i is f fee case w/ien a„ and b n do not depend on n) 
then the above expression can be written as 



(l + $(*-!)) (1 + ^-1-1)) 
^ ^ 's. 1.] 



= e l( Z + Z - 1 -2)TT^ H >1\ H ') (L12) 

fi ((i-i(--i)) (i-i^- 1 -!))) 



Note that the infinite products in these formulas are convergent because XX a « + 
Pi) < 00. 

Note also that there exists an a priori argument (see [02, §23] and [03]) ex- 
plaining why the limit functions in Theorem 1.4 factorize. This argument works for 
special values of (9, a, b) when the limit functions admit a representation theoretic 
interpretation, see §1C below. 

The proof of Theorem 1.4 is completed in section 4. The strategy of the proof is 
the same as in our paper [004]. Our main technical tool is the binomial formula for 
the Jacobi polynomials d\(z; 9, a, b) which involves the so called interpolation BC n 
polynomials Ia{x). These objects are discussed in detail in section 2 below. Note 
that the binomial formula we need is a degeneration of a more general binomial 
formula for the Koornwinder polynomials obtained in [Okl]. 

There is, however, a difference in the way we establish the sufficient conditions 
of regularity in the Jack and Jacobi cases. The argument in the Jack case relies 
heavily on the use of generating series for one-row shifted Jack polynomials. It is 
unknown how to evaluate such a series in the Jacobi case. Instead, there is a simple 
argument available which uses the stability of the polynomials 1^. 

1C. Other results. 

In section 5, we fix parameters a, b such that a > b > —\. For n = 1,2, .. . 
let Y?,' a ' b be the convex set of functions on the torus T™ which can be written as 
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convex linear combinations of functions $\(zi, . . . , z n ; 9, a, b) with £(X) < n. It 
turns out that the specialization z n — 1 determines an affine map T^'°' b — > T^'"^ . 
Consequently we can form the projective limit of the convex sets T^' a ' b as n —> oo, 
which we denote as T e . Consider the set Tg° = hniT™ whose elements arc infinite 
vectors (zi, Zi, . . . ) € T x T x . . . with finitely many coordinates Zi distinct from 1. 
Then elements of T e can be described as functions ip(zi,z 2} ■ . ■ ) on Tg° such that 
for any n = 1, 2, . . . , the function 

(fin(zi, ...,Z n ) = tp(Z!, . . . ,Z n , 1,1, . . .) 

on T" belongs to T*' a ' 6 . 

It is clear that T e is a convex set. In Theorem 5.2 we show that there is a 
one-to-one correspondence between extreme points of T e and collections (a, (3, 7) 
of VK parameters, 

a= (ai >a 2 > ••• > 0), /3 = (ft > /? 2 > • • • > 0), 7 > 0, ^(a,+A)<oo. 

Given (a,/3, 7), the corresponding extremal function on T§° has the form 

00 

^(zi,z 2 ,...) = J|^c,,/3, 7 

j=l 

where the functions <f> a ,p.-y are those defined in (1.12). That is, the possible limits of 
normalized Jacobi polynomials are precisely the extreme points of T e . In particular, 
this implies that the set T e does not depend on parameters a, b. 

In section 6, we consider 7 infinite-dimensional symmetric spaces G/K defined 
as the inductive limits of classical compact symmetric spaces G(n)/ K{n) of type 
B, C, D or BC. We assume that parameters 8,a,b take special values depending 
on the series {G(n)/K(n)}. Then the convex set T e can be identified with the 
set of positive definite, two-sided if-invariant, normalized functions on G. As a 
corollary of Theorem 5.2 we obtain an explicit description of the indecomposable 
spherical functions on G/K. Moreover, Theorem 1.4 shows how these functions 
are approximated by the indecomposable spherical functions of the rank n compact 
symmetric spaces G(n)/K(n). 

An important particular case of this result, corresponding to the large n asymp- 
totic behavior of the characters of the orthogonal and symplectic groups, was earlier 
obtained by Boyer [Boy]. 

In section 7 we present an elementary derivation of two basic facts about the 
Jacobi polynomials 3x, the binomial formula and the branching rule, for the case 
9 = 1. 

Acknowledgment. We are grateful to the referee for valuable remarks. 

2. Interpolation BC n polynomials and binomial formula 

2A. Interpolation BC n polynomials. 

Fix 9 > and denote by A^ the algebra of polynomials in n variables x\ , . . . , x TL1 
symmetric in variables Xi — 9i, i = 1, . . . , n. Consider the projective limit of these 
algebras 

A e =hjnA« 
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taken in the category of filtered algebras, with respect to homomorphisms sending 
the last variable to 0. Here the filtration is defined by the total degree deg( • ) of a 
polynomial. That is, an element / £ A* is a sequence of polynomials /„ £ A 8 such 
that /„+i(xi, . . . , x n , 0) = f n {x\, • • • , x n ) and deg(/„) remains bounded. Elements 
/ e A 8 can be evaluated at any infinite vector x = (xi, x 2 , ■ ■ ■) with finitely many 
nonzero coordinates. In particular, for any partition A, the value /(A) is well 
defined. Let h be one more variable and consider the algebras 

A e (h) = A e ®C(/i), A e [h] = A e ®C{h], 

and define A„(/i) and A^[/i] similarly. 

Proposition 2.1. (i) In the C(h)-algebra A 8 (ft,) defined above, there exist polyno- 
mials In(xi, . . . , x n ; 6; h) indexed by arbitrary partitions \i with £(/j,) < n, satisfying 
the following Newton interpolation conditions: 

(1) deg(J„(-;0;fc)) = 2R 

(2) I^ixi, . . . , x n ; 0; h) is W+-invariant in variables x^ — 6i + h, i = 1, . . . ,n, 

(3) Ifj,(X; 9;h) = if A = (Ai, . . . , A„) is a partition with < n, such that 

M % \ 

(4) I^9-h)^Q. 

(ii) These polynomials are unique up to scalar factors. 

Here the notation fi % A means that the diagram of \x is not contained in the 
diagram of A. 

Proof. Any polynomial in£i,...,i n which is W+ -invariant in variables Xi — 9i + h, 
i = 1, . ..,n, can be viewed simply as a symmetric polynomial in new variables 
(xi — 9i + h) 2 . Then the claims of the proposition become a particular case of the 
results of [Ok2] , corresponding to the case of the "perfect grid of class II" . □ 

Note that the existence of the polynomials I^(x\, . . . , x n ; 9; h) can be derived 
from the results of Okounkov [Okl] (see also Rains' paper [R] which contains a 
different approach to the results of [Okl]). Namely, 

p*( a xi a x "-a a 6 a h ~ n6 ) 
I li (xi,...,x n ;9;h) = hm ( g _ i)2H ' 

where the polynomial in the numerator is the BC n type shifted (or interpolation) 
Macdonald polynomial defined in [Okl, Definition 1.3]. 

The polynomials I fl (xi, . . . , x n ; 0; h) arc normalized by setting 

I^(p;9;h)= J] {l + ^-j + 0(^-i)){2h-l + ^+j-e(^+i)), (2.2) 

where the product is over all squares (i, j) in the diagram of fi. A motivation for 
such a normalization is given in [Ok2, Proposition 2.9] . Note that this normalization 
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is well adapted to the combinatorial formula (2.4). We shall call the polynomials 
Ip(xi, . . . , x n ; 9; h) the interpolation BC n polynomials. 

The next result provides a combinatorial formula for the interpolation BC n 
polynomials. It is convenient to state it in terms of reverse tableaux (cf. [002]). 
A reverse tableau T of shape a with entries in {1, . . . , n} is defined as a function 
T(i,j) assigning to each box (i, j) e fi a number T(i,j) e {1, . . . , n}, such that the 
numbers decrease strictly down each column and weakly from left to right along 
each row. The only difference with the conventional definition of a semistandard 
tableau consists in inverting the natural order in the set of indices 1, . . . , n. 

First, recall the combinatorial formula for the Jack polynomials with parameter 

9: 

P(h,...,j;„;0) = ^^) II x m,j)> ( 2 - 3 ) 

summed over all reverse tableaux T of shape /i, as defined above, where V>t(#) is a 
certain weight factor, which is a rational function in 9 (see formulas (7.13'), (10.10), 
(10.11), and (10.12) in chapter VI of Macdonald's book [Ma3], and also [002]; note 
that our parameter 9 is inverse to the parameter a used in [Ma3]). We do not need 
the explicit expression for ipr(9). 

Proposition 2.2. We have 

I„(x u ...,x n ;6;h) = J2M0) H + h - 9T(i, j)) 2 

t (ij)eiJ, 

-((j-i)-e(i-i) + h-eT(i,j)) 2 ], (2.4) 

summed over all reverse tableaux T of shape [i, with entries in {1, . . . ,n}, where 
iPt(9) is the same weight factor as in (2.3). 

Proof. Using (2.1), this can be obtained as a degeneration of the combinatorial 
formula for the BC n interpolation Macdonald polynomials, established in Okounkov 
[Okl, Theorem 5.2]. See also [Ok2] and Rains [R]. □ 

Note that the expression for I M (yu; 9; h) given in (2.2) can be obtained from the 
combinatorial formula (2.4). Indeed, if (x\, . . . , x n ) = (fi\, . . . , /i„) then all products 
in the right-hand side, except a single one (corresponding to a special choice of 
T) vanish, and the only nonvanishing term gives the expression (2.2). This can 
be shown using the same argument as in the second proof of Theorem 11.1 from 
Okounkov -Olshanski [001]. 

We list a number of corollaries of these two propositions. 

From the combinatorial formula (2.4) it follows immediately that the polynomials 
;0;h), which are initially defined as elements of (h) , actually belong to [h] . 
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Next, the characterization of these polynomials given in Proposition 2.1 shows 
that they are stable: 

Ip(xi, ...,x n ,0; 9; ft) = I f ,(x 1 , ...,x n ;9; ft), 

and, therefore, the sequence {I^xi, . . . , x n ; 9; ft)}„>£(^) correctly defines an cle- 
ment of the algebra A e [h}. This element will be denoted as 7 M (xi, x 2 , . . . ; 9; ft) or 
Ifj_(- ;9;h) or else simply 1^. 

Let us extend the filtration from A e to A e [ft] by setting deg h — 1. It is clear that 
the associated graded algebra gr A e [h] is naturally isomorphic to the algebra A[h], 
where A denotes the algebra of symmetric functions in infinitely many variables. 

Given an element f E A 8 [ft] , let us denote by [/] its highest degree term, which is 
a homogeneous element of the algebra A [ft]. We remark that the element € A [h] 
has degree 2 and it follows from the comparison of (2.3) and (2.4) that 

[I li (x 1 ,X2,...;6;h)]=P ll (x 1 (x 1 +2h), x 2 (x 2 + 2ft), . . . ; 9), (2.5) 

where P Jli ( • ; 9) G A is the Jack symmetric function. Note that for any homogeneous 
symmetric function g{x\, x 2 , ■ • ■ ) of degree m, the expression 

g(x\(x\ + 2ft), x 2 (x 2 + 2ft), . . . ) 

is a well defined homogeneous element of the algebra A [ft] of degree 2m. 
It is also clear that 



I^9;h) = H(»;9)(2h)M + ... (2.6) 

where the dots stand for the lower degree terms in ft and H{fi; 9) is the following 
hook-length product 

H{^9):= J] (( JM -j)-0(j i '.-i) + l). 

2B. Binomial formula. 

Here by the binomial formula we mean an expansion of the normalized Jacobi 
polynomials <J>a(zi, . . . , z n ; 9, a, b) (see (1.6)) about the point (1, . . . , 1). 

Proposition 2.3. Let £(X) < n. We have 

®\{zi,...,z n ;6,a,b) 

= IjM} + 9n) P^ Zl + z^ 1 - 2, . . . , z n + z' 1 - 2; 9) 
^ C(n,fj,;9;a,b) ' 
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where I^(X; 9;a + On) is the result of specializing h = a + 9n in I M (A; 6; h), a = 
(a + b+ l)/2, and 

C(n, ft; 6; a, b) = I^; 6;a + On) ^(l^l; 9, a, b). (2.8) 

n 

By virtue of condition 3 in Proposition 2.1, the summation in (2.7) actually goes 
on the finite set of fi's such that (iCl An explicit expression for C(n, yu; 8; a, b) is 
given in Remark 2.5 below. 

Proof. The expansion (2.7) is a limit case of the binomial formula for Koornwinder 
polynomials obtained in [Okl, Theorem 7.1]. Namely, the Koornwinder polynomials 
are orthogonal on T" with weight 

.Al n {tzf'zf'U l\ (a 1 z^,-a 2 z^, q ^ a3 z^,-q^a i zt 1 )o ' 

where g, £, ai, . . . , 0,4 are the 6 parameters and, by definition, 

00 

(ui,?i 2 ,...)oo =nn( i_ ^ ufe )- 

k i=0 

If one sets 

t = q e , 01 = g a+ 5 , a 2 = g h+ 5 , a 3 = a 4 = , 

and lets g — > 1 then these 6-parametric polynomials become 3\(z;9,a,b). Using 
this, (2.1), and the following immediate corollary of Theorem 5.2 in [Okl] 

km P;(z; q, q e ,q( a+b+1 V 2 ) = P^( Zl + zf 1 - 2, . . . , z n + z^ - 2; 6) 

one obtains (2.7) from Theorem 7.1 in [Okl]. Here P* stands for the 3-parametric 
£?C„-type interpolation Macdonald polynomial defined in [Okl, Definition 1.3]; it 
has already appeared in (2.1). 

Alternatively, one can prove (2.7) without going into g-analogs by just repeating 
the proof of main theorem of [002] and using the fact that the algebra of com- 
muting differential operators whose eigenfunctions are 3\ is isomorphic under the 
Harish-Chandra homomorphism to the algebra of polynomials in Ai, . . . , A n that 
are W + -invariant in variables 

A + p = (Ai + (n - 1)6 + cr, . . . , A„_i + 8 + a, A„ + a) . 

□ 

Note that the expansion of Jacobi polynomials in Jack polynomials was obtained 
much earlier by James and Constantine [JK] for the case when 9 = \ or 1. However, 
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in their formula the coefficients of the expansion are written in quite a different form, 
as certain combinatorial sums over standard tableaux of shape A//i (this is not 
equivalent to combinatorial formula (2.4)). Their result was extended to arbitrary 
values of 6 by Macdonald (unpublished work [Mai, §9]; see also [BeO, §5]) and 
Lassalle [L] . A discussion of the role of binomial formulas for the characters of the 
orthogonal and symplectic groups can be found in [003]. 

2C. Asymptotics of denominators in binomial formula. 

We fix an arbitrary partition p and let n go to infinity. As in Theorem 1.4, we 
assume that the parameters a, b may depend on n. We write them ciS (In i 

and 

assume that the limits (1.7) exist. 

Proposition 2.4. The denominator (2.8) in (2.7) has the following asymptotics 
C(n, p; 9; a n , h n ) ~ ^ 9^ (9 + a)^ ■ n 2 ^ , (2.9) 

where 

h(jm-,6)= [] ((w-jM(mH)+i), #'GM)= II (^i-j)-o(^-i)+e) , 

and a = lima„/n as in (1.7). 

Proof. Recall that C(n, p; 9; a n ,b n ) is the product of two terms, i^(/i; 9; a n + 9n) 
and 3^(1, ... , 1; 9, a ni 6„), where a n = (a n + b n + l)/2. We claim that, as n — > oo, 
the following two asymptotic relations hold 

I^(p; 9- a n + 9n) ~ H(p; 9) 2^ (9 + a)^n^ , (2.10) 
1 {9 + a^^ 



S.il^AaM ~ *H n H, (2.11) 

n 

where 

,. cr„ a + 6 
(T = hm — = . 

n^oo n 2 

Clearly, (2.10) and (2.11) imply (2.9). 

The first relation immediately follows from (2.6), let us check the second relation. 

The following is the general formula, due to Opdam, for the value of a multivari- 
ate Jacobi polynomial, indexed by a weight p, at the unit element, see [HS], Part 
I, Theorem 3.6.6, 

-pr r ((p + p, a v ) + k a + \k a/2 ) r ({p, a v ) + \k aj2 ) 

V{{p + p,a^) + \k a/2 ) Y{{ Pl ^) + k a + \k a/2 )' 

where a v stands for the root dual to a, and k a / 2 = if the root a/2 docs not exist. 
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In our case, the polynomial in question is just • ; 8, a n , b n ), and the unit 
element is identified with the point (1, . . . , 1). Next, we have 

p = ((n - 1)6* + a n , . . . , 8 + a n ,a n ) 

and there are 4 types of the positive roots a 

Si — Ej , £i+£j (1 < i < j < n), Si , 2ei (1 < i < n) 

with formal multiplicities 

ke z ±e, ;= 28, k Ei =a n -b n , k 2e% =b n + \. 

As the scalar product used in Opdam's formula we may take the natural scalar 
product in E™. Then the dual roots a v are as follows 

(£j ± e 3 Y = £i ± £j, £■ =2£j, (2e J ) v =e J . 

We split the product over a > into 4 products according to these 4 types of 
positive roots 

*, «„, m = n (+_) n (++) n (+) n (+2) . ^ 

n times 

where 

tt(+-) = tt T(in- K+eU-i + l)) T(6(j-i)) 

-pr(++) = yr T(fn + \ij + 0(2n - j - j + 1) + 2a n ) r(fl(2n - j - j) + 2gyQ 
11 11 T{ IH + l i j +6{2n-i-j) + 2a n ) T{6{2n - i - j + 1) + 2a n ) 

(+) -,-r T(2^ + 28(n + 2a n + 1) T(28(n - i) + 2<r n ) 



n 11^ r(2 fM + 28(n + 2a n ) T(28(n - i) + 2a n + 1) ' 

y-r(+2) _ -j-r T(n l + 8(n-i) + 2(7 n ) Y{8{n-i)+a n + \) 
~ 1 <i< n T (^ + e ( n - i ) + a n + \) T(8(n-i) + 2o n ) 

It is clear that JJ^ ' is just the formula for the value of the Jack polynomial 
(which is essentially the Jacobi polynomial for the A series). Therefore (see Stanley 
[St] or Macdonald [Ma3], VI.10.12) 



Next, we have 

n 
n 



(+) /8n + a„\ 2M {6 + a^ 2M 



8n + a n ) \ + a / 

(+2) {8n + 2<j n \ M /6> + 2ct\ H 



+ a n J \ 8 + a 
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Finally, consider the product AH factors with £(p) < i < j are trivial 

(equal to 1). Moreover, the contribution of every fixed pair i < j is asymptotically 
trivial, so that the whole contribution of the factors with 1 < i < j < £(fi) is trivial. 
Therefore, 

tt(++) tt £0fj + e ( 2n + + 2cr ») r(g(2n - i - j) + 2cr„) 

~i<ri( M ) r (^+^( 2n - + 2^) r(fl(2n-i-j + l) + 2(7 n )" 

£(Ai)<j"<ri 

For every i the product over j telescopes to 

T(m + 8{2n - i - £(fi)) + 2a n ) T(6(n - i) + 2a n ) 

T(m + 9{n + 2a n ) T(m + 9(2n - i - %)) + 2a n ) 
'26n + 2a n y i /20 + 2a AM< 



so that 



+ 2a n I \9 + 2a 



(++) / a _ \ M 



6» + 2ct 



It follows that 



n (++) n (+) n (+2) - 2M fe 



9 + a 

This gives formula (2.11) and concludes the proof. □ 

Remark 2.5. Using (2.12) one can derive the following explicit expression for the 
quantity (2.8) 

c(n,„,e,a,b) = 4 w^jl r(( n -i + i)6»)r((n-i)(? + a + i) • 

In particular, C(n, /x; 0; a, 6) actually does not depend on b. The asymptotic relation 
(2.9) is readily obtained from this expression. 

3. Sufficient conditions of regularity 

Recall that we have fixed some 9 > 0. Define elements 31,32, ■ ■ ■ in the algebra 
A of symmetric functions by means of a generating function 

00 00 
l + ^g k {x u x 2 ,...)t k = Y[(l- Xj t)- e , 

fe=i j=i 

where £ is a formal variable. These elements are algebraically independent genera- 
tors of A. Given some VK parameters a, /?, 7, define an algebra homomorphism 



e Q ,/3,7 : A — ► C 
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by 

(cf. the definition of the extended symmetric functions, see [004, Section 2.7] or 
[KOO]). In a less formal way, this homomorphism can be written as 

g (1 . ¥) -.^ e #g^. 

We will also use such a notation in the sequel. 

Proposition 3.1. Let {X(n)} be a VK sequence of partitions with parameters 
a,f3,j, see Definition 1.3, and let {h n } be a sequence of complex numbers such 
that the limit 

h = lim — 

n — >oc n 

exists. Then for any element f G A [h] we have 

f(Hn);h n ) 



rwoo n de sf ,P ' 7 V J J h=hJ 



where [/] £ A[h] is the highest degree term of f . 

Proof. Assume first that / does not depend on h, that is, / € A . Then [/] e A, 
and the claim is that 

which follows from Theorem 7.1 in [KOO] (see also Theorem 3.1 in [004]). Re- 
turning to the general case, we write / € A e [h] as a polynomial in h, 

f = fo + fih + hh 2 + ■■■ + f m h m , ft g A e , 

and apply the above formula to each f , . . . , f m . □ 
Given t G C, define a homomorphism 

7r r : A — *■ A 

by the formula 

(n T f)(xi,X2, ■ ■ •) !->■ f(xi(x-i +t),x 2 {x 2 +r), . . . ) , /GA. (3.1) 



Then we have the following 
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Proposition 3.2. 



(fr 1 | - r ^rtTT l + e(3i{T-O0i)t 

r \LL(l- Xj t)' ll(l_ a .( T + a .) t )« 



Proof. By the definition of 7r T and e ai/ 3 j7 we have 

(J (1^) = <-.*r (ft (13^; 



-w 6 



7 



^((i-^oa-^)) 6 



where t\+t 2 — rt and i x i 2 = — t , 



e 



e 



ll(l- ai ti)0 (l-a^ 2 ) e 

JlTlT1 i + e^T-e^t 



n ( i_ a . (r+a .) t ) fl 



which is the result stated. □ 

Now we can prove the following 

Theorem 3.3. Assume that the condition (1.7) is fulfilled. Let {X(n)} be a Vershik- 
Kerov sequence with parameters a, [3, 7, see Definition 1.3. For any fixed k — 
1,2, ... , ffte functions 

$ A( „ ) (z 1 , . . . ,z fe , l^^l ;0,a n ,6 n ) (3.2) 

n— fe times 



converge inftnitesimally, in the sense of Definition 1.2, to the function 

nw 5 ,(^) , (3.3) 



z=i 



where a — lim a n /n, b = \imb n /n as in (1.7), and <p a p , y a i is the following function 
of a single variable x G [—1,1] 

ft ^ 20 + 5 + 5-0/3^ 

00 1 + T flTi ( x - x ) 

w, 5 (-)=^- i) n 

i=i 



''wl eTa J {x ~ 1] 
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Proof. Apply to the functions (3.2) the binomial formula (2.7). In that formula 
fj, ranges over partitions with £(■) < n. However, in our case, the last n — k 
arguments in P fi (-;8) equal 0. By the stability of Jack polynomials, this implies 
that the summation actually goes over partitions /i with £(/i) < k. Thus, we obtain 

< E > A(n)(^i, • • • , Zk, 1, ■ ■ ■ 1 1 ; 0, tt m b n ) = 

n—k times 

^,l(p,)<k 

where ^ 

xi = — , I = 1 , . . . , k . 

Since the polynomials P^ ( • ; 6) form a homogeneous basis in the space of symmetric 
polynomials, the infinitesimal convergence of the left-hand side of (3.4) in the sense 
of Definition 1.2, as n — ► oo, is equivalent to the coefficient-wise convergence of the 
expansion in the right-hand side. Thus, we have to examine the asymptotics of the 
quantities 

I^X(n):e;a n +9n)2M , 

u fixed, n — > oo. 



C(n,^;6»;a„,6„) 
By Proposition 2.4, 

C(n, M ; 0; a n ,b n ) ~ |^ 4 W 9^ {9 + 5 )N . n 2 W , 

Next, by Proposition 3.1 we have 

rwoo n^l^l \ h=0+5 ) 

and by (2.5) and (3.1) 



which implies 



[U--,h)] = 7T 2(e+ff) P„(-;0) ) 

h=6+ij 



7 M (A(n);CT n + 6>n) 



n 



Therefore, the expansion (3.4) converges coefficient-wise to 

^.^(fl-no I ^ -^y 2/2, • • • ; (9) P, [^^y ■ ■ ■ , ^Tl) ; ' 



(3.5) 



LIMITS OF JACOBI POLYNOMIALS 



19 



where the homomorphisms e a ,0,-y an d ^2(e+s) act on the variables 2/1,2/2, • • • • The 
sum in the round brackets equals 

^^ (yi ' y2 '--- ;0) ^(#T^)'---'#T^) ;0 ) 

-nn(-^) ■ 

Here we have used the Cauchy identity for Jack polynomials (see Macdonald [Ma3, 
ch. VI, (4.13) and §10]). Therefore (3.5) equals 

n^««(n('-^)) 

Using Proposition 3.2 and the relation 2a — a + b one transforms this into the 
desired result. □ 

Corollary 3.4. In the hypotheses of Theorem 3.3, assume additionally that a n > 
b n > —\. Then for any k = 1,2, ... , the functions (3.2) converge to the function 

(3.3) uniformly on the torus T k . 

Proof. The assumption a n > b n > — i makes it possible to apply Proposition 1.1 
which implies that the functions (2.6) are positive definite functions on the torus 
T k . Since the function (3.3) is real-analytic, our claim follows from Theorem 3.3 
by virtue of a well-known general fact (see, e.g., Lemma 4.2 from [004]). □ 

4. Necessary conditions of regularity 

The argument of this section is similar to that of [004, §5]. We begin with three 
technical lemmas which then are used to prove Proposition 4.4. 

Lemma 4.1. Assume h > On — 1/2. Then we have 

J ( 2)(Ai,...,A„;0;/i) < (/(i)(Ai, . . . , A„; 6; h)) 2 

for any partition X, n= 1,2,..., and 9 > 0. 

Observe that if h = 9n + (a + b + l)/2, where a, b > —1, then the hypothesis of 
the lemma is satisfied. 

Proof. Set A = /(i)(A; 9; h) and B = i(2)(A; 0; h). From the combinatorial formula 

(2.4) we obtain 

A = J2((h + h-6i) 2 -(h-6i) 2 ) , 

i 

B = Y^ ((h + h- Oif -(h- 9if) ((A, + h- 6if - (h - 9i + l) 2 ) 

i 

<r\f\ 

+ —9 £ (fe + h 6])2 ~ {h ~ 93)2) ((At + h 01)2 -( h ~ ei+ • 

i<j 
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It is elementary to check that for any I € Z> and any i < n one has 

(l + h- Oif -{h- Oif > 
provided h > On — 1/2; in particular, 

(l + h-0i) 2 - (h-6i) 2 > 0. 

Therefore 

5 < E (( A * +h ~ 6i ) 2 -( h - 6i ) 2 ) (( A * + h - 6i ) 2 -( h ~ 6i ) 2 ) 

i 

+ ITe E (( A ^ + h - 6 tf ~( h ~ ^') 2 ) + h - Bl ? -( h - 6l ) 2 ) 

i<j 

(here we use the inequality with general I to conclude that the first factors are 
nonnegative, while the inequality with I = 1 is used to remove 1 in the second 
factors). Since < 2 for any > — 1, the lemma follows. □ 

Lemma 4.2. Suppose that 

I {1) (X(n);0;h n ) = 0{n 2 ) , n -> oo , 

and 

h n ^h n, h > 0/2, n — > oo . 

Then 

\X(n)\ = 0(n) , n — > oo . 

Observe that if h n = o n + On, where a n ~ an, a > 0, then the hypothesis of the 
lemma is satisfied. 

Proof. We have 

n 

/(i) (A; 0; h n ) = ^A,(A, + 2h n - 20i) 
i=i 

n n 

= E A '+ E ( A <- A i) + (2h n -0(n + l))J2*i- 

i—1 \<i<j<n i—1 

Since all summands are positive for large n we conclude that 

n 

(2h -0)nJ2\i(n) = 0(n 2 ) , n — > oo , 
which implies |A(n)| = 0(n). □ 
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Lemma 4.3. Let a > and b > be fixed. Assume (a, 0, 7) and (a, 0, 7) are two 
systems of VK parameters such that the corresponding functions (1.11) coincide, 



,7(*-i) 



n 



0i 2e + a + b-m\ 

1 + y — en — 



i=i fi®i( 29 + a + b + a, . 



« n 



7(x-l) 

i=l 



1 + y ( — e+i — J 

di / 2(9 + a + b + dj \ 

l -Ye{ e + l ) {x - 1] 



for any x e [—1,1]. 

Then (a,0,j) = (d,/3,7). 

Proof. A similar claim was established in step 3 of the proof of Theorem 5.1 from 
[004]. We shall use a similar argument. 
Write the above identity of functions as 



i=i [l - Aivj i=i [l - A^J 

and observe that 

oc 

ii>i 2 >--->0, Bi > B 2 > ••• > 0, ^(i,+B 4 )<°° 

i=l 

oc 

Ii>i 2 >--->0, Bi > -B 2 > ••• >0, ^(i l +B 4 )<oo 

i=l 

Since the correspondences 

dj Aj, 0i <-> Bi, &i <-► A u 0i^Bi 
are one-to-one, it suffices to prove that (4.1) implies 

= Au B, t = B u 7 = 7. 

We may extend (4.1) to an identity between two holomorphic functions in y. 
These functions are well defined at least in the left half-plane 3?y < 0. Actually, the 
left-hand side is holomorphic in the half-plane < (Ai)^ 1 and has a singularity 
at y = (Ai)^ 1 . Likewise, the right-hand side is holomorphic in the half-plane 
< an d has a singularity at y = (Ai)^ 1 . This implies A\ = A\. Thus, 
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both sides of (4.1) have common factors which can be cancelled. Iterating this 
procedure we prove that A 2 = A 2 , etc. Then we come to an identity of entire 
functions, 

oo oo 

e^JJ(l + B i y)=e^JJ(l + B i y). 

»=1 i=l 

Examining the zeros of both sides we see that Bi = B\ for all i = 1,2,..., and 
finally we conclude that 7 = 7. □ 

Theorem 4.4. Let a n > b n > —\ and let {X(n)} be a sequence of partitions with 
£(X(n)) < n. Assume that either {X(n)} is minimally regular, or the functions 

$A(n)(*, l^A;6,a n ,b n ), \z\ = 1 , (4.2) 

n— 1 times 

converge infinitesimally about z = leT 1 , see Definition 1.2. 
Then {A(n)} is a Vershik-Kerov sequence. 

Proof. The proof given below is completely parallel to the proof of Theorem 5.1 in 
[004]. 

Step 1 . Consider the following function on the unit circle T 
<p n (z) = $ A(n) (z, 1, . . . ,l;0,a n ,b n ), z ET. 

n-l 

By the binomial formula (2.7) and Proposition 2.4 we have 

<j) n {z) = 1 + A hn (z + z- 1 - 2) + A 2 . n {z + z- 1 - 2) 2 + . . . , 

where 

I (1) (\(n):6n + o- n ) I {2) (X(n);6n + a n ) 
A l n - consti , A 2 , n ~ const 2 - LJ -. , (4.3) 

with some constants not depending on n. 
We claim that 

I (1) (X(n):9n + a n ) =0(n 2 ). (4.4) 

Indeed, in case when the functions (4.2) converge infinitesimally, this bound is 
immediate. Let us prove it when {A(n)} is minimally regular. If z = e %Lp then 

z + z- 1 - 2 = 2(cos ip - 1) = -Lp 2 + ± f 4 + Oif 6 ) 

so that 

= 1 - Ai, n ip 2 + (A 2 , n + ±A hn )<p 4 + 0(<p«). 
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On the other hand, we know that <fi n (z) is a normalized positive definite function 
on T (see the proof of Corollary 3.4), hence it is the Fourier transform of a proba- 
bility measure M n on the lattice Z. The assumption of minimal regularity means 
that the measures M n weakly converge to a probability measure on Z. In such a 
situation, Lemma 5.2 from [004] says that if the second moments of the measures 
M n are not uniformly bounded then, passing to a suitable subsequence where the 
second moments tend to infinity, we obtain that the fourth moments grow faster 
than the squared second moments. Up to constant factors, the second and fourth 
moments are the coefficients in ip 2 and ip 4 , respectively. This means that if the 
number sequence is unbounded then, for a suitable subsequence of indices 

n, A 2 , n grows faster than (Ai^ n ) 2 . But, by virtue of (4.3), this contradicts Lemma 
4.1. We conclude that {^4i,n} is bounded, which is equivalent to (4.4). 

Step 2. We claim that 

\\{n)\=0{n), n^oo. (4.5) 

Indeed, this follows from the result of step 1 and Lemma 4.2. 

Step 3. Using the bound (4.5) and Cantor's diagonal process, we see that any 
subsequence of {A(n)} contains a VK subsequence. It remains to prove that any 
two VK subsequences of {A(n)} have the same VK parameters. By Theorem 3.3 
and the hypotheses of Theorem 4.4, both subsequences lead to one and the same 
limit function. Then we apply Lemma 4.3 to conclude that the VK parameters are 
the same. □ 

Proof of Theorem 1.4- Let, as usual, {X(n)} be a sequence of partitions with 
£(X(n)) < n, and let the parameters a n , b n satisfy the inequality a n > b n > —\ 
and condition (1.7). 

By Theorem 4.4, each of the 3 regularity properties of Definition 1.2 implies the 
VK property of Definition 1.3. Conversely, assume {A(n)} is a VK sequence. Then, 
by virtue of Theorem 3.3 and Corollary 3.4, this implies all 3 regularity properties 
of Definition 1.2. This proves claim (i) of Theorem 1.4. Corollary 3.4 also proves 
claim (ii) of Theorem 1.4. □ 

5. The convex set T e 

In this section we fix parameters o, b such that a > b > — |. 

For n — 1, 2, . . . let T^' a ' b be the set of functions on the torus T™ of the form 

<p(z 1 ,...,z n )= c\$ x {zi 7 ...,z n :0 7 a,b), 

\:i{\)<n 

where 

CA > 0, ° X = 1 - 

\:£(\)<n 
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Note that the coefficients c\ are uniquely determined by the function if, because 
the Jacobi polynomials form an orthogonal basis in a suitable L? space. Recall 
that each <§>\(- ;6,a,b) is a positive definite function on the torus (see the proof 
of Corollary 3.4), normalized at the unit element (1, . . . , 1) G T™. It follows that 
■ • ■ , z n ; 0, a, b)\ < 1, which implies that the series converges uniformly and 
defines a continuous function on T™. Thus, T^' a ' 6 is a subset of the set of continuous, 
positive definite, normalized functions on T™. It is clear that T^' a ' b is a convex set. 
As an abstract convex set, it is isomorphic to a simplex with infinitely many vertices. 

Proposition 5.1. Let A be an arbitrary partition with with £(X) < n. Expand the 
function 3\{zi, ■ ■ ■ , z n -i, I; 9, a, b) in Jacobi polynomials in n—1 variables Z\, ... , z n -\, 
with the same parameters 0,a,b. Then all coefficients in this expansion are non- 
negative. 

Proof. This fact can be derived by a degeneration from the branching rule for 
Koornwinder polynomials, established by Rains [R, (5.76)]. □ 

By Proposition 5.1, the specialization z n = 1 sends <$>\(zi, . . . , z n ; 6, a, b) to 
a function from T^\. Hence this specialization map determines an affinc map 
T^'°' b — > T^'"' 1 > . Using these maps for n = 2, 3, . . . we set 

T 9 =limT^, n^oo. 

As we shall see (Corollary 5.3), this projective limit space does not depend on a, b. 

An equivalent definition is as follows. Consider the set Tfj° = fimT™ whose 
elements are infinite vectors (zi, Z2, ■ ■ ■) G T xT x . . . with finitely many coordinates 
Zi distinct from 1. Then elements of T can be described as functions f{z\, Z2, ■ ■ ■ ) 
on Tg° such that for any n = 1, 2, . . . , the function 

ip n (zi, . . . , z n ) — <p(zi, ■ ■ ■ , z n , 1, 1, . . . ) 

on T" belongs to T e n ^ b . 

It is clear that T e is a convex set. Let ExT e denote the set of its extreme points. 

Theorem 5.2. There is a one-to-one correspondence between elements of the set 
ExT e and collections (a,f3, 7) of VK parameters, 

a = { ai >a 2 > ■■■ >0), /?=(/?!> /fc >■■■>()), 7>0, ^{on + ft) < 00. 

Given (a, [3, 7), the corresponding element 0/ExT 9 , viewed as a function on 
Tg° , has the form 

oc 

®a,Prf(zi, Z 2 , ■ ■ ■ ) = Y\_ <t>ot,Pa 
3 = 1 
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where (zi, Z2, ■ ■ ■ ) € Tg°, and the function (f> a ,0,f is as in Theorem 1.4, 



z + z 
2~ 



-1 



) 




OO 



(l + ±ft(z-l)) (1 + ^(z- 1 - 1)) 



((1-^-1)7(9) (1-^(^-1)7(9)) 



9 



or equivalently 



OO 



(l + ft (l-f ))(*-!) 



« [-1,1]. 



{('-tO+S))^} 



• 



i=i 



In particular, ExT 6 does not depend on the parameters a,b. 



Proof. This can be proved exactly as Theorem 1.4 in [004], see §6 in [004]. By 
virtue of a general result (see Theorem 6.1 in [004]), each function $ G ExT e can 
be approximated (uniformly on any finite-dimensional sub-torus T fe ) by a sequence 
{<£ A(n) ( ' ; 0, a , b)}- By Theorem 1.4, the limit functions 



are precisely the functions of the form $ Qjl a ;7 . This shows that Ex Y 9 C {$q./3, 7 }. 

The inverse inclusion {$ Qi /3. 7 } C ExT is obtained by a simple argument using 
de Finetti's theorem. Indeed, the fact that each function $ Qil a i7 is positive definite 
on Tg° and has the multiplicative form <j>(zi)^>(z2) ■ ■ ■ implies that $ Q ,/3, 7 is an 
extreme point in a larger convex set, namely the set of characteristic functions of 
symmetric probability measures on Z°° = Z x Z x . . . . □ 

Proposition 5.3. Any element of the convex set T s is represented by a probability 
measure on the set ExT e of extreme points, and this representation is unique. 

Proof. This claim does not follow directly from Choquet's theorem because the set 
T e is not compact. However, it can be checked, for instance, by the method of [04, 



Corollary 5.4. The set T e does not depend on the parameters a, b. 

Proof. Indeed, according to Theorem 5.2, ExT* does not depend on a, b. Then the 
corollary follows from Proposition 5.3. □ 

6. Spherical functions on infinite-dimensional symmetric spaces 

By an infinite-dimensional symmetric space we mean a homogeneous space G/K 
where the G and K are inductive limits of groups, 



$ = lim $ A(n) (-;0,a,6) 



§9]. □ 



G = fiinG(n), K = lim K(n), n — > 00, 
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such that for any n, G(n)/K(n) is a Riemannian symmetric space of compact type 
belonging to one of the classical series. We assume that rank(G (n)/K(n)) = n. 

There are 10 such spaces G/K corresponding to 10 classical series of symmetric 
spaces, see Olshanski [01], [02]: 

Table I 

1. G(n) = U(n), K{n) = 0(n). 

2. G(n) = U(n) x U(n), K{n) = U(n). 

3. G(n) = U(2n), K(n) = Sp(n). 

4. G(n) = 0(2n), K{n) = 0(n) x 0(n). 

5. G(n) = Sp{n), K(n) = U(n). 

6. G{n) = U(2n), K(n) = U(n) x U(n). 

7. G(n) = 0(n) x 0(n), K(n) = 0{n). 

8. G{n) = Sp(n) x Sp(n), K{n) = Sp(n). 

9. G(n) = Sp(2n), K = Sp{n) x Sp{n). 

10. G(n) = 0(2n), K(n) = U(n). 

Comments, a) The cmbcddings G(n) — * G(n+ 1) and K{n) — > K(n+ 1) which are 
implicit in the definition of the groups G and K are natural ones. The embeddings 
K(n) — > G(n) are also quite evident. In particular, for series 2, 7,8, these are the 
diagonal embeddings. 

b) For series 7 and 10, we wrote n instead of n because in these cases the rank 
equals [n/2]. Here one may choose one of the two possible variants: n — 2n or 
n = 2n + 1, and the inductive limit space G/K does not depend of the choice, up 
to isomorphism. 

c) For the Grassmann spaces (series 4, 6, 9), we could equally well use two distinct 
indices n\, ri2- That is, we could deal with the spaces 0(ni + n2)/0(n{) x 0(712), 
U(ni + ri2)/U{n{) x U(ri2), and Sp{n\ + 712) / Sp(n\) x Sp(ri2)- Again, such a 
generalization does affect the limit space G/K, provided that both indices go to 
infinity. 

d) In the case of series 2,7,8, the symmetric space G(n)/K(n) is one of the 
classical groups U(n), 0(n), Sp(n), and the corresponding infinite-dimensional 
space G/K coincides with one of groups 

U{oo) = limC/(n), O(oo) = limO(n), Sp(oo) = finj5p(n). 

As shown in [01], [02], for the 10 pairs (G,K) listed in Table I there is a rich 
theory of unitary representations. The present paper concerns a part of this theory 
related to spherical representations. 

Assume (G, K) is one of the 10 pairs from Table I. Let T be a unitary represen- 
tation of G in a Hilbert space H, and £ G H be a distinguished unit if-invariant 



LIMITS OF JACOBI POLYNOMIALS 



27 



vector. We say that (T, £) is a spherical representation of the pair (G, if) if £ is a 
cyclic vector. That is, if the linear span of the vectors T(g)£, where g ranges over 
G, is dense in H. Two spherical representations, (Ti,£i) and (X^,^), are said to 
be equivalent if there is an isometry Hi — > of the corresponding Hilbert spaces 
taking £1 to £2 and commuting with the action of G. 

Attached to any spherical representation (T, £) is its spherical function 

F(g) = (T(g)S,Q, 9£G. 

This is a positive definite function on the group G, two-sided invariant with respect 
to the subgroup K, and taking value 1 at the unity e e G. Denote by T(G,K) 
the set of all functions with these three properties. Then the correspondence T — > 
_F defined above determines a bijection between equivalence classes of spherical 
representations and functions from the set T(G, K). 

Of special interest are irreducible spherical representations. That is, those (T, £) 
for which T is an irreducible unitary representation of G. If T is an irreducible 
unitary representation of G then a if -invariant vector £ (provided it exists) is 
unique, within a scalar factor which does not affect the spherical function. The 
spherical functions of irreducible spherical representations are precisely the extreme 
points of T(G, K) (it is worth noting that T(G, K) is a convex set). 

Thus, classifying the irreducible spherical representations of (G, K) is equivalent 
to describing the extreme points of the convex set T(G, K). We aim to explain how 
this problem is related to that discussed in §6. 

Let T(G(n), K(n)) denote the set of functions on G(n) that are positive definite, 
two-sided if (n)-invariant and take value 1 at the unity. This is a convex set, iso- 
morphic to an infinite dimensional simplex. Note that the vertices of this simplex, 
which arc the extreme points of Y(G(n), K(n)), are the indecomposable spherical 
functions of (G(n),K(n)), that is, matrix coefficients of the form 

(T( n >(0)£< n >,£ (n) ), j£G(n), 

where is an arbitrary irreducible finite-dimensional unitary representation of 
G(n) possessing a unit K (n)-invariant vector £( n ) (such a vector is unique, within 
a scalar factor of absolute value 1). 
The natural embedding of pairs 

(G(n-l),K(n-l))^(G(n),K(n)) 

induces an affine projection 



T(G(n),K(n)) 



T(G(n - 1), K(n - 1)) 
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and we have 

T(G,K) = hmT(G(n), if (n)), n -> oo. 

Below we focus on the series 4-10, the case of series 1-3 being the subject of our 
previous paper [004]. 

So, let (G(n), K{n)) belong to one of the seven series 4-10 from Table I. Let lZ n 
denote the restricted root system of the symmetric space (G(n),K(n)). Then lZ n 
coincides with one of the classical root system B n , C n , D n or BC n equipped with 
appropriate root multiplicities. 

In all cases, it is convenient to regard lZ n as a subsystem of the BC n root system 

R n = {±ei ± e 3 ; : 1 < i < j < n} U {±e i: ±2ei : 1 < i < n} . 

To each of the series 4-10 one can attach a particular triple of parameters 0, a, b in 
such a way that the formal multiplicities k a defined in (1.5) coincide with the true 
root multiplicities in lZ n . In particular, k a = means that the root a G R n is not 
contained in 1Z n . 

Specifically, we have: 

Table II 

4. G(n) = 0(2n), K{n) = O(n) x 0(n): 9 = \, a = b = -\, K n = D n . 

5. G(n) = Sp(n), K{n) = U(n): 9 = ±, a = b = 0, K n = C n . 

6. G{n) = U(2n), K{n) = U{n) x U{n): 9 = 1, a = b = 0, 1l n = C n . 

7. G{n) = 0(n) x 0(n), K(n) = 0{n): 9 = 1, a = -\ or \, b = -\, TZ n = D n 
or B n . 

8. G{n) = Sp(n) x Sp(n), K{n) = Sp(n): 9 = 1, a = b = \, Tl n = C n . 

9. G{n) = Sp(2n), K = Sp(n) x Sp(n): 9 = 2, a = b = 1, U n = C n . 

10. G(n) = 0(2n), K(n) = U{n): 9 = 2, a = or 2, b = 0, K n = C n or BC n . 

Here, in the case of series 7 and 10, the first option for a and lZ n is chosen if 
n = 2n, and the second option is chosen if n = 2n + 1. 

Proposition 6.1. Let (G(n), K(n)) belong to one of the series J^-IO and let 9, a, b 

be the corresponding parameters as listed above. Assume additionally that the re- 
stricted root system lZ n is not D n . Then there are natural bijections 

r ^^?(G(n),K(n)), n=l,2,... (6.1) 

which are isomorphisms of convex sets and commute with the projections 

T».«.»-T£-f, T(G(n),K(n))^T(G(n-l),K(n-l)). (6.2) 

Idea of proof. Let W n denote the restricted Weyl group of the symmetric space 
G(n)/K(n). For any classical series, W n may be identified with the BC n Weyl 
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group S(n) k l? n (realized as the group W* acting on the torus T" as explained 
in §1) or with its subgroup of index 2 (the latter possibility holds exactly when 

Tin = D n ). 

It is well known that the indecomposable spherical functions of (G(n),K(n)) can 
be interpreted as normalized W n -invariant orthogonal polynomials on the torus T™ 
with the weight (1.1), where the parameters 8, a, b are those attached to the corre- 
sponding series. When 7l n ^ D n , these are precisely the normalized polynomials 
®\(zi, . . . ,z n ;6,a,b). 

This yields the required bijection (6.1). The fact that these bijections are com- 
patible with the projections (6.2) is readily verified. □ 

Corollary 6.2. Under the hypotheses of Proposition 6.1 we have: the convex set 
T(G, K) is isomorphic to the convex setT e described in §5. In particular, it depends 
only on 8 but not on a, b. 

Proof. Indeed, this follows at once from Proposition 6.1 and the results of §5. □ 

Remark 6.3. When the restricted root system 7l n is of type D n , certain polyno- 
mials $\(zi, . . . , z n ; 6, a, b) turn out to be half-sums of two distinct W„-invariant 
orthogonal polynomials. As a consequence, the set T(G(n), K(n)) turns out to be 
somewhat larger than the set Y n ' a ' b . Nevertheless, the claim of Corollary 6.2 holds 
in this case as well. Indeed, as is seen from the list above, the equality lZ n = D n 
occurs in two cases: for series 4 and for series 7 with n even. In the latter case we 
may choose n odd without changing the limit space G/K. In the former case the 
same effect is achieved if we take G{n) = 0(2n + to), K(n) = 0(n + to) x O(n), 
where to is an arbitrary fixed positive integer. 

7. The BC n polynomials with 8 = 1 

When 6 = 1, the multivariate Jacobi polynomials 3x and the interpolation poly- 
nomials J M admit explicit dcterminantal expressions. This makes it possible to 
establish the basic facts about these polynomials independently of the general the- 
ory, in a rather elementary way. 

Throughout the present section we assume 8 = 1 and fix arbitrary parameters 
a > — l,b > — 1. We give explicit formulas for both kinds of polynomials and sketch 
elementary proofs of the binomial formula (2.7) and of the branching rule for Jacobi 
polynomials, which in turns implies Proposition 5.1. 

Let pi(x; a, b) (where I = 0, 1, 2, . . . ) denote the classical Jacobi polynomials in a 
single variable x, orthogonal on the segment — 1 < x < 1 with the weight function 
(1 - x) a (l + x) b . We use the same normalization as in Erdelyi et al. [Er]. An 
explicit expression for pi(x; a, b) in terms of the Gauss hypergeometric function 2 Fi 
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is 

Ma;; a ' h) = r(^%r(a + i) zFlH ' l + a + b + 1 > a + 1 -> t 5 )- ( 7 - x ) 

In particular, the value at a; = 1 is given by 

r(/ + a + l)) 



Pi0-;a,b) = 



r(/ + i)r(o + i) 

and the leading coefficient in pi(x; a, b) is 

n m o-l ^(21 + a + b+l) 

*fr°' 6 > = 2 r(/ + a + b + OT + i) - 

More generally, for any n = 1,2,... and any partition A with £(X) < n we set 

det [pA i+ „-i(a;j ;a,6)] 
y x (x 1 ,...,x n ;a,b) = ^±= ^ , (7.2) 

where 

n»o= II (xi-xj). 

\<i<j<n 

Proposition 7.1. T/ie BC„ orthogonal polynomials with 6 = 1 and arbitrary pa- 
rameters a,b> —1 are expressed through the polynomials (7.2) as follows 

3x(zi, ■ ■ ■ , z n ; 1, a, 6) = const <£ A ^— , • ■ • , — ; f ■ ^ 

w/iere 

2l A l 

const 



lir=i + n - z; a, 6) 

This fact is undoubtedly well known. E.g., in an equivalent form, it was pointed 
out in Lassalle [L]. For reader's convenience we present a proof. 

Proof. It is readily verified that the polynomials (7.2) arc pairwise orthogonal on 
the n-dimensional cube [—1, 1]™ with respect to the measure 

V 2 (x) J] {±-Xi) a {l + Xi) b dxi...,dx n 

l<i<n 

which implies that the polynomials 



are pairwise orthogonal on the torus T" with the weight (1.1) specialized at 6 = 1. 
Next, we have 

%i\(xi, . . . , x n ; a, b) = const' x\ 1 . . . x* n + ■ ■ ■ 
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where dots mean lower terms in lexicographic order and 

n 

const' = J^J +n — i;a, 6), 
i=l 

which implies the triangularity condition (1.2) for the polynomials on the torus, 
defined by the right-hand side of (7.2). 

The stronger triangularity condition (1.4) can also be readily verified. Indeed, 
comparing the determinantal expression (7.2) with the determinantal formula for 
the Schur polynomials 

det 

, . _ l<i,j<n 

S^yXl, ■ • ■ , Xn) — ^ 

we see that 

5Pa(£i, ■ ■ -,x n ;a,b) = ax^s^xi, ...,x n ) 

MCA 

where fj, C A means that the diagram of fj, is contained in that of A, and are 
certain coefficients. Next, recall the well-known triangularity property of the Schur 
polynomials: 

^ju(^l > ■ ■ ■ ; ^n) ^ ^ K^uTYlv {X\ , . . . , X n ), 

where m v is the conventional monomial symmetric function, K nu are certain coeffi- 
cients (the Kostka numbers) and v < fi is the dominance order on partitions {[i — v 
can be written as a linear combination of the vectors — Sj, i < j). 

From the last two formulas we obtain the triangularity condition of the form 
(1.4): 

fz! + 1 z n + Z- 1 \ ^ ( zi+ Zy 1 z„ + z" 1 \ 

v 7 y<A v 7 

= X! C A^( Z 1 . •• - ,2n) 

with certain coefficients b\ v and ca^- 

Thus, the polynomials in [z\, . . . , z n ) defined by the right-hand of (7.2) possess 
the characteristic properties of the BC n orthogonal polynomials on the torus T™ 
with parameter 9 — 1 and hence coincide with the polynomials 3\{zi, ■ ■ ■ , z n ; 1, a, b). 

□ 

By virtue of (7.2), Proposition 7.1 provides an explicit determinantal expression 
for the polynomials 3\(zi, ■ ■ ■ , z n ; 1, a, b). Now we shall give an explicit expression 
of the BC n interpolation polynomials with 9 = 1. 



Hi~\-n—i 
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We need a notation. Given an infinite sequence of parameters A = (A\ , A 2 , . . . ) , 
define "generalized powers" of a variable y by 

{y\A) m = {y-A l )...(y-A m ), m=l,2,...; {y \ A)° = 1. 

Next, we define (generalized) factorial Schur polynomials in n variables by 

det Uyi | A) ^ +"-■?'] 

/ i * \ Ki,j<n ' _ N 

A = ~ T/ / x (7-3) 

where /i is an arbitrary partition with £(p.) < n. Note that s^(yi, . . . ,y n \ A) is an 
inhomogeneous symmetric polynomial whose top degree homogeneous component 
is the conventional Schur polynomial s M (yi, . . . , The polynomials (7.3) share 
many properties of the conventional Schur polynomials. A number of formulas for 
the polynomials (7.3) can be found in Macdonald [Ma2] and [Ma3, Example 1.3.20], 
Molev [Mo], and also, for the special case A = (0, 1, 2, . . . ), in Okounkov-Olshanski 
[001]. Note that our notation for the parameters Ai,A2,... differs from that of 
Macdonald by sign. 

Proposition 7.2. The BC n interpolation polynomials with 9 — 1 are expressed 
through the factorial Schur polynomials (7.3) as follows 

7 M (xi, ...,x n ;l;h) = s p ((xi + h - l) 2 , . . . , (x n + h - n) 2 \ A), (7.1) 

where 

A=((h- n) 2 , (h-n+ l) 2 , (h-n + 2) 2 , . . . ). 

Proof. We have to check that in the special case 9 = 1, all claims of Proposition 2.1 
hold and the corresponding polynomials coincide with those given by formula (7.3). 
This was shown, in an elementary way, in Okounkov-Olshanski [003, Theorem 
2.5]. □ 

About this result, see also [Ok2, §3.3]. 

Proposition 7.3. The polynomials I^(xi, . . . ,x n ; 1; h) defined by (7.1) satisfy the 
combinatorial formula of Proposition 2.2. That is, 

I li (x 1 ,...,x n ;l;h)=^2 II [( x t(i,]) + h - T(i, j)) 2 

-{j-i + h-T(i,j)) 2 ] , 
summed over all reverse tableaux T of shape fi, with entries in {1, ... ,n}. 
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Proof. This is a special case of the combinatorial formula for factorial Schur poly- 
nomials, 

S»(yi, ...,y n \A) = ^2 II (yT(i,j) - A j-i+n+l-T(i,j)), 

summed over all reverse tableaux T of shape //, with entries in {1, . . . , n}. About 
the latter formula, see Goulden-Greene [GG] , Macdonald [Ma2] , and also Okounkov 
[Ok2, §3.3]. □ 

The binomial formula (2.7) of Proposition 2.3 reduces to the following claim 
Proposition 7.4. Let £(X) < n. We have 

<Pa(zi, ...,x n ;a,b) ^ J M (A; l;a + n) s^xi -l,...,x n -l) 



= y 

«Pa( 1, a, b) c(n,n;a) 



where / M (A; l;a + n) is the result of specializing h = a + n in I M (A; l;h), a = 
(a + b+l)/2, and 

l)r(m+n-i + a+l) 



T(n-i + l)T(n-i + a+l) 



Proof. By virtue of the definition of the polynomials • ; 1, h) this formula can be 
rewritten as 

%S\(xi, . . . ,x n ;a,b) 



*p A (L__^ 1 l;a,6) 

n 

= E 



s^jll ...,11\<t 2 ,(o- + l) 2 , {o- + 2) 2 , . . . ) SjAxi -l,...,x n -l) 

c(n, /i; a) 



where 

h = Aj + n — i + a, i = 1, . . . , n. 

The latter formula can be directly derived from the determinantal formula (7.2) and 
the expression (7.1) for the classical Jacobi polynomials. For some special values 
of the parameters a, b (which correspond to characters of classical groups of the B, 
C, D series) such a computation was done in Okounkov-Olshanski [003, Theorem 
1.2]. For general a, b the argument is quite similar. □ 

The next result is a special case of Proposition 5.1. 
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Proposition 7.5. Let n — 2,3,... and X be a partition with £(X) < n. In the 
expansion 

qS\(xi,...,x n -i,l;a,b)= ^ (■ ■ ■ ) Vi^( x i, ■ ■ ■ , x n -i; a, b) 

v. t(v)<n-l 

all coefficients (...) are nonnegative. 

Proof, a) First, let us describe the scheme of the proof. We shall use certain 
renormalized polynomials 

R\(xi, . . . , x n ) — (a positive factor) Cp>(xi, . . . , x n ; a, b) 

(7.4) 

Rn{x\, . . . ,x n -i) = (a positive factor) *p A {xi , . . . ,x n -i;a + 1, b). 

Below \i and v denote partitions with £(■) < n—1. We shall establish the following 
two-step branching rule, which implies the proposition: 

R\(xi, . . .,x n -i, 1) = ^ R^(x!, . . .,x n -i) (7.5) 

R tl {xi,...,x n -i)= ^2 A(v,,v)R v (x 1 ,...,x n - 1 ), A(fi,v)>0, (7.6) 

where fi -< A means 

Ai > Hi > A 2 > • • • > A n _i > n n -i > A„ 

and v -< fi U means 

Mi > v\ > fi 2 > ■ ■ ■ > Mn-l > ^n-l > 0. 

The coefficients A(/j,,v) are specified below, see (7.13) and (7.14). 

The proof presented below is quite elementary (we found it in 1998). It is 
worth noting that a much more general two-step branching rule, in the context of 
Koornwinder polynomials, was established by Rains, see [R, (5.76)]. 

b) Now we proceed to the proof of the proposition. For I = 0, 1, 2, . . . set 

n(*) = ££*4 r l{X ) = n+l{X) - n{X) . (7-7) 
pi(l;a,b) x-l 

We have 

ri(x) — (a positive factor) pi(x; a, b), 

(7-8) 

n{x) = (a positive factor) pi(x; a + 1,6). 

Indeed, the first relation in (7.8) is evident, because pi(l;a, 6) > 0. Let us check 
the second relation. 
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Since n + i(l) = n(l) = 1, it is clear that ri(x) is a polynomial, and its degree is 
strictly equal to /. We have 



1 



x m n(x)(l - x) a+i (l + x)»dx 

-1 

x m r l+1 (x)(l -x) a (l+x) b dx+ f x m n(x)(l-x) a (l+x) b dx, 



and the latter two integrals vanish whenever m < n. This implies that ri(x) is 
proportional to pi(x; a + 1,6). 

By the definition of n(x), its leading coefficient is the same as that of ri+i(x), 
hence positive. This completes the proof of (7.8). 

c) Set 

^det. [r Xi +n-i(xj)] 

RX{XU ■ ■ ■ ' Xn) = ~ l V(x u ...,x n ) (7 - 9) 

...det [r^+ n -i-t(x 3 )} 
i? Al (xi,...,a;„_ 1 ) = (7.10) 

K(ori, . . . ,X n -l) 

Then, due to (7.8), we have (7.4). 

d) Let us check the first step of our branching rule, (7.5). By virtue of (7.9), the 
left-hand side of (7.5) is given by the ratio of a determinant of the order n and a 
Vandermonde. Examine the determinant in the numerator. For i = 1, . . . ,n — 1, 
let us subtract the i + 1th row from the zth row. Then we come to a determinant 
of order n — 1. Its (i, j)th entry is equal to 

r\ i+n -i(xj) - r\ i+1+n -i-i(xj), 

and we may divide it by (xj — 1), because of the obvious relation 

n-l 

V{xi, . . .,a; n _i, 1) = V(ari, . . . ,a;„_i) JJ(a;j - 1). 
Then the entry will take the form 



n 1+ i{x ) -n 2 {x ) _ ^ 

1 m=l 2 



where we abbreviated Zi = Aj + n — i — 1, Z2 = K+i + n — I — i. 

Employing the latter expression and expanding the determinant along the rows 
we get the desired result (7.5). 

e) Let us check the second step of the branching rule, (7.6). 
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Write the three-term recurrence relation for the polynomials r m : 

r m+ i(x) = (a m x + b m )r m (x) - c m r m -i(x), m > 1. (7-11) 

By the normalization, 

a m + On 1- 

By making use of these relations and the definition of r m we get 

Vm O'mXm ~t~ c m^m — 1? 771 ^ 1. 

Iterating this relation we further get 

m 

r TO = -B(m, l)ri +c m ... c k r k -i, m>k>0, (7-12) 

l=k 

where 

c p I ai , m > I > 0. 

y;<p<m y 

When k = we agree that r_i = so that the last term in (7.12) disappears. 
By virtue of (7.9) and (7.10), the relation (7.6) is equivalent to 

, det \r^+n-i-t(x 3 )} = A ^,v) det [r„ 1+n _i_i(a;j)] 

l<2,7<n— 1 * — * Ki.i<ii-1 

Examine the determinant in the left-hand side. Its entries in the ith row are of the 
form r^ i+n -i-i(xj), where j = 1, ... ,n — 1. We shall apply to them the relation 
(7.12), taking m = /i i +n — 1 — i, k = /i^+i + n — 1 — i. 

First, do this for the first row, 7 = 1. We get a decomposition of the form 

r^ +n - 2 {xj) = (-)n(zj) + (...)?V 2+ „-3(zj), 

i=(U2+n— 2 

where the coefficients marked as (...) are expressed through the c- and a-coefficients. 
Remark that the last term coincides, within a scalar factor, with that in the second 
row. Consequently, when we expand the determinant along the first row, it will 
play no role. 

Now, we perform this expansion and then look at the second row and repeat the 
same procedure, etc. For the «th row (i = 1, . . . , n — 2) we get the decomposition 

/^i+n— 1 — i 

T^+n-l-iiXj) = ^ {■■■)n{Xj) + {...)f lli+1+n -2-i{Xj), 

l—fii^x+n—l — i 
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and when we come to the last row (i = n — 1) then we choose k = so that the "last 
term" mentioned above will disappear at all. Finally we get the desired expression 
(7.6) with coefficients A(fi, v) given by 

n-i 

v ) = TT B i^i +n-l-i, Ui +n-l-i). (7.13) 

4=1 

e) To check that these coefficients are strictly positive we use a well-known 
general property of orthogonal polynomials: the coefficients a m , c m in the three- 
term relation (7.11) are strictly positive provided that the leading coefficients of 
the polynomials are strictly positive (see [Er], section 10.3, formulas (7) and (8)). 
The latter property holds in our case, so that we conclude that B(m, I) > for any 
m > I > 0, and finally v) > 0. 

f) The coefficients B(m, I) entering the formula (7.13) can be explicitly computed: 

B(m,l) 

_ (2m + a + b)T(m + b + l)m\(2l + a + b+ l)T(l + a + b+ l)T(l + a + 1) 
~ 2T(m + a + b + 2)r(m + a + 2)T(n + b+ Tjl\ ~(7l±) 

When / = 0, the product (21 + a + b + l)T(l + a + b + 1) must be replaced by 
T(l + a + b+2) = T{a + b + 2). 

Indeed, let k m be the leading coefficients in p m (x;a, b) and h m be the squared 
norm of p m (x;a,b) (this is the standard notation, see [Er, §10.3]), and set also 
e m = p m (l;a,6). It follows from [Er, §10.3 (8)] and (7.7) that 

_ ^m+l^-m ^m— 1 ^m^m— 1 

771 — ~~U ' ^ m — / 2 h 5 

whence 

h , h p 2 

B(m, I) — 



Using the explicit values of the constants entering this formula (see [Er, §10.8]) we 
get (7.14). □ 
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